Painting into Music: An interactive multimedia edutainment tool for painting

Interactivity is an integral part of engaging children with what they are learning. We address this engagement by creating an interactive entertainment and learning tool for children to use in the classroom or at home using a painting scenario. We do this by coalescing music, multimedia and electronics together through the real-time mapping of painting gestures into musical output and feedback. The system analyses and objectifies aspects of the painting process, measuring painting gestures with multiple sensors and using the data to create a meaningful musical feedback that is controlled by the actions of the painter on the interactive “canvas”. The system integrates various software modules, sensor inputs and hardware systems. The paper describes several different mapping strategies, from simple one to one relationships, to complex webs of interaction between gestural data and sonic processing and controls. Input features include brush stroke analysis such as speed and position, shape and angles of drawn items. They can be configured and mapped to offer different mapping strategies, focusing on varied characteristics of the painting gesture, depending on the application context. Keywords— multimedia; interactive; painting; mapping; sensor